In most computer systems, a substantial portion of the system hardware and processing time involves data transfers between processors and memory and transfer of instructions from memory to processors. Computers often employ a hierarchy of memory with various trade offs between access time and cost. In particular, the most recently accessed data and instructions may be copied into an especially fast memory called a cache memory. For high performance, a cache memory for instructions is especially important. If the next instruction is in the cache memory, the processor can typically immediately execute the next instruction. If the next instruction is not in the cache memory (called a cache "miss"), the processor may have to pause while the next instruction is fetched from a slower memory elsewhere in the memory hierarchy. For high performance, there is a need to minimize cache misses and to minimize any delay in case of a cache miss.
Compared to the main memory, a cache memory is typically smaller, faster, more expensive, and physically closer to the processor. An integrated circuit processor may have the cache memory within the integrated circuit. In general, for performance, the cache memory needs to be large to minimize cache misses. Integrated circuit space, however, is limited so that a large on-chip cache is often impractical. There is a need for an instruction cache system providing high performance within a minimal amount of on-chip space.